Image processor and image processing method

ABSTRACT

An image processor according to the present embodiment includes an acquirer, an output unit, and a generator. The acquirer acquires a first pixel value obtained by making a pixel value in a first image captured in a first exposure time correspond to a predetermined sensitivity and a second pixel value obtained by making a pixel value in a second image captured in a second exposure time shorter than the first exposure time correspond to the predetermined sensitivity. The output unit outputs a pixel value serving as a predetermined value larger than the second pixel value when the first pixel value has been saturated and the second pixel value corresponding to the first pixel value is less than the first pixel value. The generator generates a high dynamic range image using at least the first image and the second image based on the pixel value output by the output unit.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a divisional of application Ser. No. 15/459,604,filed Mar. 13, 2017 and is based upon and claims the benefit of priorityfrom the prior U.S. Provisional Patent Application No. 62/307,837, filedon Mar. 14, 2016, the entire contents of which are incorporated hereinby reference.

FIELD

The present invention relates to an image processor and an imageprocessing method.

BACKGROUND

To obtain a high dynamic range image serving as an image representing awide dynamic range, a High Dynamic Range Imaging (HDRI) for synthesizinga long exposure image and a short exposure image captured in a shorterexposure time than the long exposure image has been performed. In theshort exposure image, a flicker, a variation in pixel value depending onan image capture timing, can occur. Thus, in the HDRI, processing forreducing an effect of the flicker is required.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example of a configuration ofan image processor according to an embodiment;

FIG. 2 is a diagram illustrating a relationship between a sub-pixel andan exposure time;

FIG. 3 is a diagram illustrating a relationship between illuminance oflight irradiated onto a sub-pixel and a pixel value;

FIG. 4 is a diagram illustrating a relationship between an exposure timeof a sub-pixel and an illumination pattern;

FIG. 5 is a diagram illustrating a relationship between an illuminationpattern and a pixel value;

FIG. 6 is a diagram illustrating a determination table used when twotypes of images respectively captured in different exposure times aresubjected to HDRI processing;

FIG. 7 is a flowchart illustrating HDRI processing according to thedetermination table;

FIG. 8 is a flowchart illustrating HDRI processing performed for threetypes of images respectively captured in different exposure times;

FIG. 9 is a flowchart illustrating HDRI processing performed when alight emitting diode (LED) light source is captured; and

FIG. 10 is a flowchart illustrating processing for determining a flickerand a blur using three types of images respectively captured indifferent exposure times.

DETAILED DESCRIPTION

According to an aspect of the present invention, an image processorincludes an acquirer, an output unit, and a generator. The acquireracquires a first pixel value obtained by making a pixel value in a firstimage captured in a first exposure time correspond to a predeterminedsensitivity and a second pixel value obtained by making a pixel value ina second image captured in a second exposure time shorter than the firstexposure time corresponds to the predetermined sensitivity. The outputunit outputs a pixel value serving as a predetermined value larger thanthe second pixel value when the first pixel value has been saturated andthe second pixel value corresponding to the first pixel value is lessthan the first pixel value. The generator generates a high dynamic rangeimage using at least the first image and the second image based on thepixel value output by the output unit.

Hereinafter, embodiments of the present invention will be described withreference to the drawings. The embodiments do not limit the presentinvention.

First Embodiment

An image processor according to an embodiment acquires a first pixelvalue obtained by making a pixel value in a long exposure imagecorrespond to a reference sensitivity and a second pixel value obtainedby making a pixel value in a short exposure image correspond to thereference sensitivity, and performs HDRI using the first pixel valuewhen the first pixel value is saturated and the second pixel value isless than the first pixel value, to reduce an effect of a flickeroccurring in the short exposure image. More details will be describedbelow.

FIG. 1 is a block diagram illustrating an example of a configuration ofthe image processor 1 according to an embodiment. The image processor 1is an apparatus which performs HDRI, and includes an image capture unit100, a corrector 102, an acquirer 104, a synthesizer 106, a determiner108, an output unit 110, and a generator 112.

The image capture unit 100 captures an image in a plurality of differentexposure times. The image capture unit 100 is an image sensor in whichpixels are arranged in a two-dimensional or planar shape, for example,and each of the pixels includes four sub-pixels. The sub-pixels havesimilar structures and different exposure times. Each of the sub-pixels,for example, consists of a photo-diode as its main component. That is,respective sensitivities of the four sub-pixels are determined dependingon the exposure times. As the exposure time increases, the sensitivityincreases. The four sub-pixels are a sub-pixel having a long exposuretime (hereinafter referred to as a long exposure sub-pixel), a sub-pixelhaving a medium exposure time 2 (hereinafter referred to as a mediumexposure 2 sub-pixel), a sub-pixel having a medium exposure time 1(hereinafter referred to as a medium exposure 1 sub-pixel), and asub-pixel having a short exposure time (hereinafter referred to as ashort exposure sub-pixel) in descending order of the sensitivities.Thus, four different pixel values are obtained from the same pixel.

Thus, when the image capture unit 100 performs image capturing, an imagecaptured in a long exposure time (hereinafter referred to as a longexposure image), an image captured in a medium exposure time 2(hereinafter referred to as a medium exposure 2 image), an imagecaptured in a medium exposure time 1 (hereinafter referred to as amedium exposure 1 image), and an image captured in a short exposure time(hereinafter referred to as a short exposure image) are obtained indescending order of sensitivities. The image capture unit 100 is usedfor a vehicle camera and a mobile camera, for example. The imageprocessor 1 performs HDRI using at least two of the four imagesrespectively captured in different exposure times.

The medium exposure 2 sub-pixel(an LED pixel) is a dedicated lightemitting diode (LED) sub-pixel. That is, the medium exposure 2 sub-pixelwhich takes into consideration the characteristics of an LED lightsource. The LED sub-pixel is a pixel corresponding to characteristicssuch as illuminance of the LED light source and a flashing frequency(100 Hz or 120 Hz). For example, an exposure time having a lengthsufficient to suppress the occurrence of flicker is set in considerationof flashing frequency.

The corrector 102 is connected to the image capture unit 100, andcorrects respective pixel values of a plurality of images obtained bythe image capture unit 100 to corrected pixel values corresponding to areference sensitivity. The reference sensitivity is the sensitivity ofthe long exposure sub-pixel. Thus, the corrector 102 performs thecalculation: (corrected pixel value)=(correction coefficient)×(pixelvalue before correction). The correction coefficient is expressed by anequation: (correction coefficient)=(reference sensitivity)/(targetsensitivity). The four sub-pixels constituting each of the pixels in theimage capture unit 100 respectively have similar configurations, andeach of the reference sensitivity and the target sensitivity is a valueproportional to the length of the exposure time. Thus, the correctioncoefficient has a value equal to an exposure magnification. Therespective exposure times of the four sub-pixels are an exposure timeunder long exposure (hereinafter referred to as a long exposure time)TL, an exposure time under medium exposure 2 (hereinafter referred to asa medium exposure 2 time) TM2, an exposure time under medium exposure 1(hereinafter referred to as a medium exposure 1 time) TM1, an exposuretime under short exposure (hereinafter referred to as a short exposuretime) TS in descending order of the exposure times.

According to the above, the corrector 102 first performs thecalculation: (exposure magnification under medium exposure 2(hereinafter referred to as medium exposure 2 magnification) Rm2)=(longexposure time TL)/(medium exposure 2 time TM2), the calculation:(exposure magnification under medium exposure 1 (hereinafter referred toas medium exposure 1 magnification) Rm1)=(long exposure time TL)/(mediumexposure 1 time TM1), and the calculation: (exposure magnification undershort exposure (hereinafter referred to as a short exposuremagnification) Rs)=(long exposure time TL)/(short exposure time TS). Ifthe exposure time TL is 12 ms, and the exposure time TS is 6 ms, forexample, Rs=2.

Then, the corrector 102 performs the calculation: (corrected pixelvalue)=(exposure magnification)×(pixel value before correction) and(exposure magnification)=(correction coefficient). A pixel value in thelong exposure image, a pixel value in the medium exposure 2 image, apixel value in the medium exposure 1 image, and a pixel value in theshort exposure image are respectively indicated by L, M2, M1, and S. Therespective corrected pixel value are respectively indicated by L′, M2′,M1′, and S′. For example, the corrected pixel value S′ in the shortexposure image is obtained as (corrected pixel value S′)=(exposuremagnification Rs)×(pixel value S in short exposure image).

The acquirer 104 is connected with the corrector 102, and acquires thecorrected pixel value L′, M2′, M1′, S′. That is, the acquirer 104acquires the corrected pixel value obtained by making the pixel value ineach of the plurality of images correspond to the reference sensitivity.Since the long exposure magnification RL is 1, the corrected pixel valueL′ is equal to the pixel value L in the long exposure image.

The corrector 102 may be provided in the image capture unit 100.Alternatively, the corrector 102 may be provided in the acquirer 104.

The synthesizer 106 is connected to the acquirer 104, and acquires acomposite value using any one of the corrected pixel values L′, M2′,M1′, and S′. That is, the synthesizer 106 calculates a composite valueobtained by synthesizing the plurality of corresponding corrected pixelvalues at a predetermined ratio. When a predetermined ratio a is takenas 0.5, for example, a composite value G′ is calculated by an equation:(composite value G′ (L′, S′))=(1−α)×(corrected pixel valueL′)+α×(corrected pixel value S′).

The composite value may be calculated by an equation: (composite valueG′ (L, S))=(1−α)×F1(pixel value L)+α×F2(pixel value S). F1(L) and F2(S)are respectively functions of the pixel value L and the pixel value S.That is, the composite value may be calculated using any one of thepixel values L, M2, M1, S. For example, a may be changed according tothe pixel value L.

The determiner 108 is connected to the acquirer 104. The determiner 108determines which of the corrected pixel values L′, M2′, M1′, and S′ andthe composite pixel value G′ is to be output depending on any one of thecorrected pixel values L′, M2′, M1′, and S′ and the composite pixelvalue G′.

The output unit 110 is connected to the acquirer 104, the synthesizer106, and the determiner 108, and outputs a pixel value according to thedetermination by the determiner 108. That is, the output unit 110outputs any one of the corrected pixel values L′, M2′, M1′, and S′ andthe composite pixel value G′.

The generator 112 is connected to the output unit 110, and generates ahigh dynamic range image using at least two of the long exposure, themedium exposure 2, the medium exposure 1, and the short exposure. Theimage generated is output to a monitor, a printer, or the like. Thegenerator 112 may perform image processing such as frequency processingand hierarchy conversion processing for the generated image. The monitormay display a moving image or display a still image using a plurality ofimages obtained by HDRI.

The respective exposure times of the four sub-pixels constituting thepixel in the image capture unit 100 will be described below withreference to FIG. 2. FIG. 2 illustrates a relationship between asub-pixel and an exposure time. A horizontal axis represents an exposuretime, and a vertical axis represents the type of sub-pixel. Each of thesub-pixels starts image capturing at a time T0. Then, when an exposuretime TS elapses, the short exposure sub-pixel stops image capturing.Then, when an exposure time TM1 elapses, the medium exposure 1 sub-pixelstops image capturing. When an exposure time TM2 elapses, the mediumexposure 2 sub-pixel stops the image capturing. When an exposure time TLelapses, the long exposure sub-pixel stops image capturing. Thus, thesub-pixels are configured to simultaneously start and stops imagecapturing in ascending order of the exposure times. The light exposuretime may be called an exposure time or a shutter speed. The foursub-pixels may have different exposure start times.

FIG. 3 illustrates the relationship between illumination of lightirradiated onto a sub-pixel and a pixel value. A horizontal axisrepresents illuminance, and a vertical axis represents a pixel value. Asolid line represents a pixel value, and a broken line represents acorrected pixel value.

As illustrated in FIG. 3, the pixel value L increases in proportion tothe illuminance, and is saturated when the illuminance is not less thanSL. The pixel value M2 also increases in proportion to the illuminance,and is saturated when the illuminance is not less than S2. The pixelvalue M1 also increases in proportion to the illuminance, and issaturated when the illuminance is not less than S1. Similarly, the pixelvalue also increases in proportion to the illuminance, and is saturatedwhen the illuminance is not less than SS.

And, corrected pixel value M2′, M1′, and S′ are corrected to correspondto the exposure time of the long exposure sub-pixel. Thus, each of thecorrected pixel values has the same increase rate to an increase rate ofthe pixel value L against the illuminance until saturated.

As can be seen from the foregoing, in image capturing with light havingilluminance of less than SL, any of four images has not been saturated.In this illuminance range, respective contrasts and Signal-to-Noise(S/N) ratios of the long exposure image, the medium exposure 2 image,the medium exposure 1 image, and the short exposure image decrease inthis order. Thus, the long exposure image is mainly used for HDRI in theimage processor 1.

Then, when the illuminance is not less than SL and less than S2, themedium exposure 2 image, the medium exposure 1 image, and the shortexposure image have not been saturated. In this illuminance range, therespective contrasts and S/N ratios of the medium exposure 2 image, themedium exposure 1 image, the short exposure image decrease in thisorder. Thus, the medium exposure 2 image is mainly used for HDRI.

Then, when the illuminance is not less than SL and less than S1, themedium exposure 1 image and the short exposure image have not beensaturated. In this illuminance range, the respective contrasts and S/Nratios of the medium exposure 1 image and the short exposure imagedecrease in this order. Thus, the medium exposure 1 image is mainly usedfor HDRI.

Then, when the illuminance is not less than S2 and less than SS, theshort exposure image has not been saturated. In this illuminance range,the short exposure image is mainly used for HDRI.

In the image capture unit 100, the image capturing may be performed fourtimes by changing the exposure time. In this case, each of the pixelscan be composed of one sub-pixel. Similar sub-pixels or differentsub-pixels may be respectively used for the four sub-pixels. If thedifferent sub-pixels are used, an aperture ratio may be changed for eachof the sub-pixels, or a material may be changed for each of thesub-pixels. Alternatively, an amplification factor may be changed foreach of the sub-pixels.

A flicker appearing in the short exposure image will be described belowwith reference to FIGS. 4 and 5. FIG. 4 illustrates a relationshipbetween an exposure time of a sub-pixel and an illumination pattern P1,P2, P3, and P4. A horizontal axis represents an exposure time, and avertical axis represents the type of exposure length. Like in FIG. 2,the short exposure sub-pixel starts image capturing at time T0, and endsthe image capturing after a lapse of an exposure time TS. The longexposure sub-pixel starts image capturing at time T0, and ends the imagecapturing after a lapse of an exposure time TL.

The illumination pattern P1 is continuously irradiated onto each of thesub-pixels at illuminance K1. The illuminance K1 is illuminance at whichthe long exposure sub-pixel is not saturated. The illuminance isschematically indicated by the thickness of a line. The illuminationpattern P2 is continuously irradiated onto each of the sub-pixels atilluminance K2. At the illuminance K2, the short exposure sub-pixel isnot saturated, and the long exposure sub-pixel is saturated.

The illumination patterns P3 and P4 respectively schematically representillumination patterns of the LED light source in a traffic light, forexample. That is, the illumination patterns P3 and P4 periodically flashat illuminance equal to the illuminance K2. The periodic illuminationpattern is approximately 100 Hz or approximately 120 Hz, for example,and looks like continuous light to the naked eye.

The illumination pattern P3 starts to be irradiated simultaneously witha timing T0 at which the short exposure sub-pixel starts imagecapturing. On the other hand, the illumination pattern P4 starts to beirradiated at a timing before the end of the exposure time TS.

Then, respective pixel values and corrected pixel values obtained whenthe illumination patterns P1, P2, P3, and P4 illustrated in FIG. 4 areirradiated onto each of the sub-pixels will be described below. FIG. 5is a diagram illustrating a relationship between an illumination patternand a pixel value. A horizontal axis represents illuminance, and avertical axis represents a pixel value. A solid line represents a pixelvalue, and a broken line represents a corrected pixel value.

Respective illuminances of the illumination patterns P1, P2, P3, and P4illustrated in FIG. 4 are indicated by arrows P1, P2, P3, and P4 in FIG.5. A square L1 illustrated in FIG. 5 indicates a pixel value in theillumination pattern P1 captured by the long exposure sub-pixel, and S1′indicates a corrected pixel value in the illumination pattern P1captured by the short exposure sub-pixel. The illuminance of theillumination pattern P1 is less than SL. Therefore, the pixel value L1in the long exposure image has not been saturated. Thus, the pixel valueL1 and the corrected pixel value S1′ substantially become equal to eachother.

A square L2 indicates a pixel value in the illumination pattern P2captured by the long exposure sub-pixel, and S2′ indicates a correctedpixel value in the illumination pattern P2 captured by the shortexposure sub-pixel. The illuminance of the illumination pattern P2 isnot less than SL. Therefore, the pixel value L2 in the long exposureimage is saturated, and becomes equal to a saturated value. On the otherhand, the illuminance of the illumination pattern P2 is less than SS.Therefore, the corrected pixel value S2′ in the short exposure image hasnot been saturated. Thus, the corrected pixel value S2′ is more than thepixel value L2.

Furthermore, a square L3 indicates a pixel value in the illuminationpattern P3 captured by the long exposure sub-pixel, and S3′ indicates acorrected pixel value in the illumination pattern P3 captured by theshort exposure sub-pixel. The illumination pattern P3 is irradiated witha sufficient amount of light to saturate the long exposure sub-pixel,although flashing. Thus, the pixel value L3 in the long exposure imagerepresents a saturated value.

Furthermore, the illumination pattern P3 is periodically flashing.Therefore, the corrected pixel value S3′ becomes less than the correctedpixel value S2′. On the other hand, a ratio of overlapping of anexposure time and an irradiation time of the short exposure sub-pixelbecomes large, as illustrated in FIG. 4. Thus, the corrected pixel valueS3′ becomes more than the pixel value L3, as illustrated in FIG. 5.

A square L4 indicates a pixel value in the illumination pattern P4captured by the long exposure sub-pixel, and S4′ indicates a correctedpixel value in the illumination pattern P4 captured by the shortexposure sub-pixel. The pixel value L4 in the long exposure imagerepresents a saturated value, like in the illumination pattern P3.

However, the illumination pattern P4 is only irradiated during a part ofthe exposure time of the short exposure sub-pixel, as illustrated inFIG. 4. Thus, the corrected pixel value S4′ becomes less than thesaturated pixel value L4, as illustrated in FIG. 5.

As can be seen from the foregoing, as the pixel value in the shortexposure image, the corrected pixel values S3′ and S4′ vary depending onan image capture timing. In this case, the pixel values S3 and S4 alsovary. Thus, an image which looks as if an LED light source disappeared,for example, may be obtained depending on an image capture timing. Ashort exposure image looks as if it were flashing when viewed in amoving image. Such a phenomenon is referred to as a flicker. A flicker,which appears when the LED light source is captured, is referred to asan LED flicker phenomenon.

If the illumination pattern P2 is continuously irradiated during theexposure time TS, and the pixel value L2 has been saturated, thecorrected pixel value S2′ always becomes more than the pixel value L2.Similarly, if the illumination pattern 2 is continuously irradiated atilluminance made lower than K2, when the pixel value in the longexposure image is saturated, the corrected pixel value S2′ in the shortexposure image becomes more than the saturated pixel value L2 in thelong exposure image.

Thus, a combination of the corrected pixel value S3′ and the saturatedpixel value L3 each obtained in the illumination pattern P3 is alsoobtained even by continuous irradiation at illuminance made lower thanK2. In normal shooting, the illuminance of light irradiated onto thesub-pixel is unclear. Thus, when the corrected pixel value S3′ is notless than the saturated pixel value L3, it cannot be determined whetherthe illumination pattern P3 has been irradiated during a part of theexposure time TS only under this condition.

On the other hand, the corrected pixel value S4′ becomes less than thesaturated pixel value L4, as obtained in the illumination pattern P4,only when the illumination light has been irradiated during a part ofthe exposure time TS. Thus, when the pixel value L4 in the illuminationpattern P4 is saturated, and the corrected pixel value S4′ in theillumination pattern P4 is less than the saturated pixel value L4, theillumination light has been irradiated only during a part of theexposure time TS.

Thus, as the exposure time decreases, a flicker more easily occurs. Ingeneral HDRI, if the pixel value L has been saturated, a high dynamicrange image is generated primarily using the corrected pixel value S′.Thus, in general HDRI, an image, which looks as if a light sourceperiodically flashing, like the LED light source, disappeared dependingon a shooting timing, is generated.

Then, an example of HDRI processing for reducing an effect of such aflicker and obtaining a high dynamic range image will be describedbelow. FIG. 6 illustrates a determination table used when two types ofimages respectively captured in different exposure times are subjectedto HDRI processing. This determination table is used by the determiner108 when HDRI processing is performed using the pixel value L and thecorrected pixel value S′, for example. If two images respectivelycaptured in different exposure times are acquired, the image captured inthe longer exposure time is a long exposure image (first image), and theimage captured in the shorter exposure time is a short exposure image(second image). The length of the exposure time is not limited.

The determiner 108 in FIG. 6 determines whether the pixel value L hasbeen saturated (YES or NO), and determines whether the corrected pixelvalue S′ is less than a corresponding pixel value L (YES or NO). Adetermination result is output as signals 0, 1, and 2 to the output unit110.

In the determination table, when the determination result is the signal0, it is determined that a flicker has occurred in the corrected pixelvalue S′. That is, if the pixel value L is saturated, and the correctedpixel value S′ corresponding to the pixel value L is less than the pixelvalue L, it is determined that a flicker has occurred in the correctedpixel value S′.

Thus, the determiner 108 determines whether a flicker has occurred usingthe pixel value L and the corresponding corrected pixel value S′. If thepixel value L is a larger than or equal to predetermined value, e.g.,1023 or if the corrected pixel value S′ is a predetermined value, thedeterminer 108 determines that the pixel value has been saturated.

A state where a flicker has occurred means a state where a sub-pixel isilluminated from a light source which flashes and a pixel value outputby the sub-pixel varies depending on an image capture timing. A casewhere a flicker has occurred in a pixel value means a case where a pixelvalue obtained by actual image capturing is less than a pixel valueobtained by continuous irradiation using the same illuminance, that is,a state where a ratio of an illumination time of a light source whichflashes to a exposure time of a sub-pixel used to capture the lightsource and obtain a pixel value is less than a predetermined value.

FIG. 7 is a flowchart illustrating HDRI processing according to adetermination table. The corrector 102 acquires a pixel value L in along exposure image (first image) and a pixel value S in a shortexposure image (second image) from the image capture unit 100 (S700).Then, the corrector 102 calculates a corrected pixel value L′ of thepixel value L and a corrected pixel value S′ of the pixel value S(S702). Since a exposure magnification RL is 1, the corrected pixelvalue L′ is equal to the pixel value L.

Then, the determiner 108 determines whether the pixel value L has beensaturated (S704). If the pixel value L is or over a predetermined value,e.g., 1023, the determiner 108 determines that the pixel value L hasbeen saturated. If the pixel value L has not been saturated (NO inS704), the determiner 108 then outputs a signal 2 to the output unit110, and outputs a composite value G′ (L, S′) from the output unit 110(S706). This composite value G′ (L, S′) is calculated by an equation:(composite pixel value G′ (L′, S′))=(1−α)×(pixel value L)+α×(correctedpixel value S′), where α=0.2, for example, in the synthesizer 106. Ifthe pixel value L has not been saturated, an S/N ratio and a contrast ofa long exposure image are respectively higher than an S/N ratio and acontrast of a short exposure image, and thus a synthesis ratio of thepixel value L is increased.

On the other hand, if the pixel value L has been saturated (YES inS704), the determiner 108 determines whether the corrected pixel valueS′ is less than the pixel value L (S708). If the corrected pixel valueS′ is not less than the pixel value L (NO in S708), the determiner 108outputs the corrected pixel value S′ from the output unit 110 (S710). Ifthe corrected pixel value S′ is less than the pixel value L (YES inS708), the determiner 108 then outputs a signal 0 to the output unit110, and outputs the pixel value L from the output unit 110 (S712), andthe processing ends. While a flow of processing corresponding to onepixel has been described in FIG. 6, the same processing is alsoperformed for other pixel values constituting a long exposure image anda short exposure image. In a process in step S712, if a pixel valuelarger than the corrected pixel value S′ is output, an effect of aflicker can be reduced. Thus, a pixel value, which is not less than asaturated pixel value of the pixel value L and is less than a saturatedpixel value of the corrected pixel value S′, for example, may be outputfrom the output unit 110. In this case, the pixel value to be outputfrom the output unit 110 may be a constant or a variable value.

If the pixel value L has not thus been saturated, the determiner 108outputs a composite value G′ (L, S′), to improve an image quality at lowilluminance. On the other hand, if the pixel value L has been saturated,and the corrected pixel value S′ is not less than the pixel value L, thecorrected pixel value S′ is output. Thus, a higher dynamic range imagethan that when only the long exposure image is used can be obtained.

Furthermore, if the pixel value L has been saturated, and the correctedpixel value S′ corresponding to the pixel value L is less than the pixelvalue L, the pixel value L is output. That is, if the determiner 108determines that a flicker has occurred in the corrected pixel value S′,the determiner 108 outputs the pixel value L from the output unit 110.The determination processing prevents a pixel value to be output to theoutput unit 110 from being less than the pixel value L. Thus, even if aflicker has occurred in the pixel value S, an effect on an image afterHDRI processing is reduced. Thus, an effect of the flicker can bereduced, and a high dynamic range image can be obtained.

Another example of HDRI processing for reducing an effect of a flickerwhile obtaining a high dynamic range image will be described below. FIG.8 is a flowchart illustrating HDRI processing using three types ofimages respectively captured in different exposure times. HDRIprocessing using a long exposure image (first image), a medium exposure1 image (third image), and a short exposure image (second image) will bedescribed below. The HDRI processing differs from the processingillustrated in FIG. 7 by also using the medium exposure 1 imagetherefor.

The corrector 102 acquires a pixel value L in the long exposure image, apixel value M1 in the medium exposure 1 image, and a pixel value S inthe short exposure image from the image capture unit 100 (S800). Then,the corrector 102 calculates a corrected pixel value M1′ of the pixelvalue M1 and a corrected pixel value S′ of the pixel value S (S802).

Then, the determiner 108 determines whether the pixel value L has beensaturated (S804). If the pixel value L has not been saturated (NO inS804), the determiner 108 then outputs a composite value G′ (L, M1′)from the output unit 110 (S806). This composite value G′ (L, M1′) iscalculated by an equation: (composite pixel value G′ (L,M1′))=(1−α)×(pixel value L)+α×(corrected pixel value M1′), where α=0.3,for example, in the synthesizer 106.

On the other hand, if the pixel value L has been saturated (YES inS804), the determiner 108 determines whether the corrected pixel valueM1′ has been saturated (S808). If the corrected pixel value M1′ has notbeen saturated (NO in S808), the determiner 108 determines whether thecorrected pixel value M1′ is less than the pixel value L (S810). If thecorrected pixel value M1′ is not less than the pixel value L (NO inS810), the determiner 108 outputs a composite value G′ (M1′, S′) fromthe output unit 110 (S812). This composite value G′ (M1′, S′) iscalculated by an equation: composite pixel value G′ (M1′,S′))=(1−α)×(corrected pixel value M1′)+α×(corrected pixel value S′),where α=0.3, for example, in the synthesizer 106. On the other hand, ifthe corrected pixel value M1′ is less than the pixel value L (YES inS810), the determiner 108 outputs the pixel value L from the output unit110 (S814).

On the other hand, if the corrected pixel value M1′ has been saturated(YES in S808), the determiner 108 determines whether the corrected pixelvalue S′ is less than the pixel value L (S816). If the corrected pixelvalue S′ is not less than the pixel value L (NO in step S816), thedeterminer 108 outputs the corrected pixel value S′ from the output unit110 (S818). On the other hand, if the corrected pixel value S′ is lessthan the pixel value L (YES in S816), the determiner 108 outputs thepixel value L from the output unit 110 (S820), and the processing ends.

If the pixel value L has not thus been saturated, the determiner 108outputs a composite value G′ (L, M1′), in which the ratio of the pixelvalue L is increased, to the output unit 110. If the pixel value L hasbeen saturated, and the corrected pixel value M1′ has not beensaturated, the determiner 108 outputs a composite value G′ (M1′, S′), inwhich the ratio of the pixel value M1′ is increased, from the outputunit 110. Thus, a composite value having a high S/N ratio and a highcontrast can be output depending on illuminance.

If the pixel value L has been saturated, and the corrected pixel valueM1′ corresponding to the pixel value L is less than the pixel value L,the determiner 108 outputs the pixel value L from the output unit 110.Thus, even if a flicker has occurred in the pixel value M1 in the mediumexposure 1 image, an image can be generated using the pixel value L.Thus, an effect of the flicker on the processed image can be reduced.

Even if a flicker has occurred in the corrected pixel value S′, an imagecan be generated using the pixel value L, like in the processingillustrated in FIG. 6. Thus, even if a flicker occurs in either one of ashort exposure image and a medium exposure 1 image, an effect of theflicker can be reduced. Thus, a high dynamic range image is generatedwhile a flicker such as an LED flicker can be reduced occurring in aprocessed image.

FIG. 9 is a flowchart illustrating HDRI processing performed when an LEDlight source is captured. The HDRI processing differs from theprocessing illustrated in FIG. 8 by using a long exposure image, amedium exposure 2 image, and a short exposure image therefor. Asub-pixel, which has captured the medium exposure 2 image, as describedabove, is a sub-pixel for an LED having an exposure time adjusted tomatch characteristics of the LED light source. A case where the LEDlight source such as a traffic light is included in a shooting target,and HDRI is performed mainly using the medium exposure 2 image will bedescribed.

The corrector 102 acquires a pixel value L in a long exposure image(first image), a pixel value M2 in a medium exposure 2 image (fourthimage), and a pixel value S in a short exposure image (second image)from the image capture unit 100 (S900). Then, the corrector 102calculates a corrected pixel value L′ of the pixel value L and acorrected pixel value S′ of the pixel value S (S902). The sensitivity ofthe sub-pixel, which has captured the medium exposure 2 image, is takenas a reference sensitivity. In this case, an exposure magnification RLis less than 1. Therefore, the corrected pixel value L′ becomes lessthan the pixel value L.

Then, the determiner 108 determines whether the pixel value M2 has beensaturated (S904). If the pixel value M2 has not been saturated (NO inS904), the determiner 108 then outputs the pixel value M2 to the outputunit 110 (S906).

On the other hand, if the pixel value M2 has been saturated (YES inS904), the determiner 108 determines whether a composite value G′ (L′,S′) is less than the pixel value M2 (S908). The composite value G′ (L′,S′) is calculated by an equation: (composite pixel value G′ (L′,S′))=(1−α)×L′+α×S′, if L′ is not less than w′, α=1.0. On the other hand,if L′ is less than w′, α=L′/w′. Thus, if the composite value G′ (L′, S′)has a similar characteristic to that of the corrected pixel value S′,and a flicker has occurred in the corrected pixel value S′, thecomposite value G′ (L′, S′) becomes less than the pixel value M2.

If the composite value G′ (L′, S′) is not less than the pixel value M2(NO in S908), the determiner 108 outputs the composite value G′ (L′, S′)from the output unit 110 (S910). On the other hand, if the compositevalue G′ (L′, S′) is less than the pixel value M2 (YES in S908), thedeterminer 108 outputs the pixel value M2 to the output unit M2 (S912),and the processing ends.

If the pixel value M2 has not thus been saturated, the determiner 108outputs the pixel value M2 to the output unit 110. Since the LED lightsource is a main shooting target, the pixel value M2, which has beenadjusted to an LED characteristic, is preferentially used. If a flickerhas occurred in the corrected pixel value S′, the composite value G′ (L,S′) becomes less than the pixel value M2. Therefore, when the pixelvalue M2 is output to the output unit 110, an effect of the flicker canbe reduced. If the pixel value M2 has been saturated, and a flicker hasnot occurred in the corrected pixel value S′, the composite value G′ (L,S′) can be output to the output unit 110. The corrected pixel value S′accounts for a main percentage of the composite value G′ (L, S′).Therefore, a high dynamic range image having a wider dynamic range canbe generated. Thus, the high dynamic range image is generated while aneffect of an LED flicker can be reduced.

An example of processing for determining a flicker and blur will bedescribed below. FIG. 10 is a flowchart illustrating processing fordetermining a flicker and a blur using three types of imagesrespectively captured in different exposure times. HDRI processing usinga long exposure image (first image), a medium exposure 1 image (thirdimage), and a short exposure image (second image) will be describedbelow. Similar processes to those illustrated in FIG. 8 are assigned thesame reference numerals, and hence description thereof is not repeated.

The determiner 108 determines whether a difference between a correctedpixel value S′ and a corrected pixel value M1′ is within a firstpredetermined rang of value (S1000). If the difference between thecorrected pixel value S′ and the corrected pixel value M1′ is within thefirst predetermined rang of value (YES in S1000), the determiner 108determines whether a difference between the pixel value L and thecorrected pixel value S′ or the corrected pixel value M1′ exceeds asecond predetermined value (S1002). If the difference between the pixelvalue Land the corrected pixel value S′ or the corrected pixel value M1′exceeds the second predetermined value. (YES in S1002), the determiner108 determines that a blur has occurred (S1004). On the other hand, ifthe pixel value L is the same as the positive pixel value S′ or thecorrected pixel value M1′ (NO in S1002), the determiner 108 determinesthat a situation is normal, i.e., neither a flicker nor a blur hasoccurred (S1006).

On the other hand, if the corrected pixel value S′ and the correctedpixel value M1′ are not the same (NO in S1000), the determiner 108determines that a flicker has occurred (S1008). The determiner 108determines that a flicker has occurred on the premise that the correctedpixel value M1′ has not been saturated. While the determinationprocessing has been performed using the pixel value L, the pixel valueM1, and the pixel value S within the same pixel, the present inventionis not limited to this. For example, determination processing may beperformed using an average value of long exposure pixels, an averagevalue of medium exposure pixels, and an average value of short exposurepixels within a predetermined range.

If the corrected pixel value S′ and the corrected pixel value M1′ arethe same, the determiner 108 can determine whether a blur has occurredor a situation is normal by comparing the pixel value L with thecorrected pixel value S′ or the corrected pixel value M1′. Thus, it canbe determined that the pixel value L in the long exposure image has beensaturated without comparing the pixel value L with the corrected pixelvalue. If the corrected pixel value S′ and the corrected pixel value M1′differ from each other, it can be determined that a flicker has occurredon the premise that the corrected pixel value M1′ has not beensaturated.

As described above, according to the present embodiment, if the pixelvalue L in the long exposure image has been saturated, and the correctedpixel value S′ in the short exposure image is less than the pixel valueL, when HDRI is performed using a predetermined pixel value larger thanthe corrected pixel value S′, an effect of a flicker can be reduced, anda high dynamic range image can be obtained.

At least a part of the image processor 1 in the above embodiments may beformed of hardware or software. In the case of software, a programrealizing at least a partial function of the image processor 1 may bestored in a recording medium such as a flexible disc, CD-ROM, etc. to beread and executed by a computer. The recording medium is not limited toa removable medium such as a magnetic disk, optical disk, etc., and maybe a fixed-type recording medium such as a hard disk device, memory,etc.

Further, a program realizing at least a partial function of the imageprocessor 1 can be distributed through a communication line (includingradio communication) such as the Internet. Furthermore, this program maybe encrypted, modulated, and compressed to be distributed through awired line or a radio link such as the Internet or through a recordingmedium storing it therein.

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the inventions. Indeed, the novel methods and systems describedherein may be embodied in a variety of other forms; furthermore, variousomissions, substitutions and changes in the form of the methods andsystems described herein may be made without departing from the spiritof the inventions. The accompanying claims and their equivalents areintended to cover such forms or modifications as would fall within thescope and spirit of the inventions.

1. An image processor comprising: an acquirer configured to acquire afirst pixel value obtained by making a pixel value in a first imagecaptured in a first exposure time correspond to a predeterminedsensitivity and a second pixel value obtained by making a pixel value ina second image captured in a second exposure time shorter than the firstexposure time correspond to the predetermined sensitivity; an outputunit configured to output a predetermined value larger than the secondpixel value when the first pixel value has been saturated and the secondpixel value corresponding to the first pixel value is less than thefirst pixel value; and a generator configured to generate a high dynamicrange image using at least the first image and the second image based onthe predetermined value output by the output unit.
 2. The imageprocessor according to claim 1, wherein the output unit outputs thefirst pixel value as the predetermined value.
 3. The image processoraccording to claim 1, wherein the output unit outputs a pixel value,which is not less than a saturated pixel value of the first pixel valueand less than a saturated pixel value of the second pixel value, as thepredetermined value.
 4. The image processor according to claim 1,wherein the predetermined sensitivity is a value based on the firstexposure time, the acquirer acquires the pixel value in the first imageas the first pixel value, and the acquirer acquires as the second pixelvalue a pixel value obtained by subjecting the pixel value in the secondimage to correction for making the first exposure time and the secondexposure time corresponds to each other.
 5. The image processoraccording to claim 1, further comprising a determiner configured todetermine whether the first pixel value has been saturated and determinewhether the second pixel value is less than the first pixel value whenthe first pixel value has been saturated.
 6. The image processoraccording to claim 5, wherein the determiner determines that the firstpixel value has been saturated when the pixel value in the first imageis or over the predetermined value or when the first pixel value is orover the predetermined value.
 7. The image processor according to claim1, wherein the output unit outputs the second pixel value when the firstpixel value has been saturated and the second pixel value is not lessthan the first pixel value.
 8. The image processor according to claim 1,further comprising a synthesizer configured to synthesize the firstpixel value and the second pixel value in a predetermined ratio toobtain a composite value, wherein the output unit outputs the compositevalue when the first pixel value has not been saturated.
 9. The imageprocessor according to claim 1, wherein the acquirer further acquires athird pixel value obtained by making a pixel value in a third imagecaptured in a shorter exposure time than the first exposure time andcaptured in a longer exposure time than the second exposure timecorrespond to the predetermined sensitivity, and the output unit outputsthe first pixel value when the first pixel value and the third pixelvalue have been saturated and the second pixel value is less than thefirst pixel value.
 10. The image processor according to claim 9, whereinthe output unit outputs the second pixel value when the first pixelvalue and the third pixel value have been saturated and the second pixelvalue is not less than the first pixel value.
 11. The image processoraccording to claim 9, wherein the output unit outputs a composite valueobtained by synthesizing the second pixel value and the third pixelvalue in a predetermined ratio when the first pixel value has beensaturated and the third pixel value is not less than the first pixelvalue.
 12. The image processor according to claim 1, wherein theacquirer further acquires a fourth pixel value obtained by making apixel value in a fourth image captured in a shorter exposure time thanthe first exposure time and captured in a longer exposure time than thesecond exposure time correspond to the predetermined sensitivity, andthe output unit outputs the fourth pixel value when the fourth pixelvalue has not been saturated.
 13. The image processor according to claim12, further comprising a synthesizer configured to synthesize the firstpixel value and the second pixel value in a predetermined ratio toobtain a composite value, wherein the output unit outputs the fourthpixel value when the fourth pixel value has been saturated and thecomposite value is less than the fourth pixel value.
 14. The imageprocessor according to claim 13, wherein the output unit outputs thecomposite value when the fourth pixel value has been saturated and thecomposite value is not less than the fourth pixel value.
 15. The imageprocessor according to claim 1, further comprising a determinerconfigured to determine that a flicker has occurred in the second pixelvalue when the first pixel value is saturated and the second pixel valuecorresponding to the first pixel value is less than the first pixelvalue.
 16. The image processor according to claim 1, wherein theacquirer further acquires a third pixel value obtained by making a pixelvalue in a third image captured in a third exposure time shorter thanthe first exposure time and longer than the second exposure timecorresponds to the predetermined sensitivity, and further comprising adeterminer configured to determine that a blur has occurred when adifference between the second pixel value and the third pixel value iswithin a predetermined range of value and a difference between the firstpixel value and the second pixel value exceeds a predetermined value.17. An image processing method comprising: acquiring a first pixel valueobtained by making a pixel value in a first image captured in a firstexposure time correspond to a predetermined sensitivity and a secondpixel value obtained by making a pixel value in a second image capturedin a second exposure time shorter than the first exposure timecorrespond to the predetermined sensitivity; outputting a pixel valueserving as a predetermined value larger than the second pixel value whenthe first pixel value has been saturated and the second pixel valuecorresponding to the first pixel value is less than the first pixelvalue; and generating a high dynamic range image using at least thefirst image and the second image based on the pixel value output by theoutput unit.
 18. The image processing method according to claim 17,further comprising determining that a flicker has occurred in the secondpixel value when the first pixel value has been saturated and the secondpixel value corresponding to the first pixel value is less than thefirst pixel value.
 19. The image processing method according to claim17, wherein the acquiring comprises further acquiring a third pixelvalue obtained by making a pixel value in a third image captured in athird exposure time shorter than the first exposure time and longer thanthe second exposure time correspond to the predetermined sensitivity,and further comprising determining that a blur has occurred when thesecond pixel value and the third pixel value are equal to each other andthe first pixel value differs from the second pixel value.